Chapter 9 - Decision trees
Chapter 10 - Bagging

Terminology

Regression versus classification

Regression trees predict the average response value in a subgroup; classification trees predict the class that this observation belongs to

How deep?

Preventing overfitting

• Restrict tree depth
• Restrict minimum number of observations in terminal nodes
• Pruning: make a complex tree first, simplify afterwards

Bias and variance

Prerequisites

# Helper packages
library(dplyr)       # for data wrangling
library(ggplot2)     # for awesome plotting

# Modeling packages
library(rsample)     # for sampling the data 
library(rpart)       # direct engine for decision tree application
library(caret)       # meta engine for decision tree application
library(ipred)       # bagging

# Model interpretability packages
library(rpart.plot)  # for plotting decision trees
library(vip)         # for feature importance
library(pdp)         # for feature effects

ames <- AmesHousing::make_ames()
set.seed(123)
split <- initial_split(ames, prop = 0.7,
                     strata = "Sale_Price")
ames_train  <- training(split)
ames_test   <- testing(split)

Ames housing example

ames_dt1 <- rpart(
  formula = Sale_Price ~ .,
  data    = ames_train,
  method  = "anova")

Feature interpretation

Automated feature selection: uninformative features are not used in the model.

Feature interpretation

Decision trees: pros

• Easy to explain, visually appealing

• Require little preprocessing

• Not sensitive to outliers or missing data

• Can handle mix of categorical and numeric features

Decision trees: cons

• Not the best predictors (other models we’ve seen so far are better at predicting)

• Simple yes/no questions result in rigid, non-smooth boundaries

• Deep trees: low bias, high variance (risk of overfitting)

• Shallow trees: high bias, low variance (low predictability)

Bagging: bootstrap aggregating

• Fit multiple prediction models and take the average

• By model averaging, bagging helps to reduce variance and minimize overfitting

• Especially useful for unstable, high variance models (where predicted output undergoes major changes in response to small changes in the training data)

Bagging - method

• Create b bootstrap copies of the original training data

  • Bootstrapping: make new training sets by taking random samples with replacement

• Fit your algorithm (commonly referred to as the base learner) to each bootstrap sample

• New predictions are made by averaging predictions of the individual base learners

Bagging examples

Bagging 50-500 decision trees leads to optimal performance

Implementation

• A single pruned decision tree performs worse than MARS or KNN

• 100 unpruned, bagged decision trees perform better

• Depending on number of iterations, this can become computationally intense

  • But: iterations are independent so easy to parallelize

Packages: {ipred} and {caret}

# make bootstrapping reproducible
set.seed(123)

# train bagged model (ipred package)
ames_bag1 <- ipred::bagging(
  formula = Sale_Price ~ .,
  data = ames_train,
  nbagg = 100,  
  coob = TRUE,
  control = rpart.control(minsplit = 2, cp = 0)
)

ames_bag2 <- caret::train(
  Sale_Price ~ .,
  data = ames_train,
  method = "treebag",
  trControl = trainControl(method = "cv", number = 10),
  nbagg = 200,  
  control = rpart.control(minsplit = 2, cp = 0)
)
ames_bag2

How many trees?

Feature importance

Feature importance

Bagging: pros and cons

Pro

• Bagging improves prediction accuracy for high variance (and low bias) models

  • but: at expense of interpretability and computational speed

Con

• Tree correlation: despite bootstrapping, many trees will be similar (esp. at the top)